AI now beats humans at basic tasks — new benchmarks are needed, says major report

🌈 Abstract

The article discusses the rapid progress in artificial intelligence (AI) systems, such as the chatbot ChatGPT, which are now matching or exceeding human performance in various tasks. It highlights the key findings from the Artificial Intelligence Index Report 2024, published by the Institute for Human-Centered Artificial Intelligence at Stanford University. The report emphasizes the need for new ways to assess AI systems as current benchmarks are becoming obsolete due to the fast-paced advancements in the field.

🙋 Q&A

[01] Rapid Progress in AI

1. What are some of the key findings from the Artificial Intelligence Index Report 2024 regarding the progress in AI systems?

• AI systems like ChatGPT are now matching or exceeding human performance in tasks such as reading comprehension, image classification, and competition-level mathematics.
• The pace of progress in machine learning systems has been "startlingly rapid," with benchmarks becoming obsolete within a few years, compared to 5-10 years in the past.
• The report highlights the need for new ways of assessing AI, such as evaluating their performance on complex tasks like abstraction and reasoning.

2. How is the growing use of AI in science highlighted in the report?

• The report dedicates an entire chapter to science applications of AI, highlighting projects like Graph Networks for Materials Exploration (GNoME) from Google DeepMind, which aims to help chemists discover new materials, and GraphCast, another DeepMind tool for rapid weather forecasting.

3. What is the current state of AI development in terms of industry versus academia?

• The industry sector produced 51 notable machine-learning systems last year, while academic researchers contributed 15.
• Academic work is shifting towards analyzing the models coming out of companies and developing tougher tests to assess the capabilities of large language models (LLMs).

[02] Challenges and Concerns

1. What are the concerns regarding the costs and energy use of AI systems?

• The costs of training AI models like GPT-4 and Google's Gemini Ultra are extremely high, reaching tens of millions of dollars.
• There are concerns about the energy use and water consumption needed to cool the data centers that run these AI systems, which are described as "very inefficient."

2. What are the ethical concerns surrounding the development and use of AI?

• There is growing international divide, with some countries being very excited about AI and others being very pessimistic.
• In the United States, there has been a steep rise in regulatory interest, with the number of AI-related bills proposed by policymakers increasing significantly after 2022.
• Regulatory action is increasingly focused on promoting responsible AI use, but the lack of standardized assessments for responsible use makes it difficult to compare the risks posed by different AI systems.

3. What are the concerns about the availability of training data for AI systems?

• The report notes that some researchers are now worried about running out of high-quality language data, with the non-profit research institute Epoch projecting that this could happen as soon as this year (although their latest analysis suggests 2028 is a better estimate).